Multi-lingual conference bridge with cues and method of use

ABSTRACT

The present invention relates to a new method and system for use of a multi-protocol conference bridge, and more specifically a new multi-language conference bridge system and method of use where different cues, such as an attenuated voice of an original non-interpreted speaker, is used to improve the flow of information over the system.

FIELD OF THE INVENTION

The present invention relates to a new method and system for use of amulti-lingual conference bridge, and more specifically a newmulti-language conference bridge system and method of use wheredifferent cues and time stamps are used to further improve the easy flowof information over the bridge.

BACKGROUND

For decades, certain institutions, such as the United Nations, or largeinternational corporations with offices around the world have conductedbusiness in multiple languages. When physical meetings at the UN areconvened, delegates in an auditorium in view of other delegates speak intheir native languages. Interpreters are present and interpret what issaid into the languages of the other delegates.

As a delegate speaks, the interpreter speaks in another language, theinterpretation of what was said by the first delegate. To make themeeting as dynamic, interactive and productive as possible, theinterpretation is often ideally done “simultaneously.” This means thatspecially trained individuals listen to a delegate speak in onelanguage, and interpret as rapidly as possible over the flow of words inwhat is called simultaneous interpretation. The delay of experiencedinterpreters can be reduced to several seconds or even less, dependingon the languages involved.

Delegates in such a meeting are equipped with microphones and headphonesto hear interpretations. The interpreters can be isolated in soundproofbooths and are also equipped with microphones and headsets. A venue isspecially wired and controls are provided to delegates, interpreters andmoderators that allow for selection of audio inputs and outputs andcontrol of audio mixing electronics. These semi-manual systems areextremely complex and costly.

In contrast to meeting physically at an auditorium, increasingly,organizations (including companies but also governments, non-profits,various regulatory, rule-making and standards bodies) convene their“meetings” using conference call technology as a way to avoid the timeand expense of travel. Delegates or employees can participate from theirhome locations over a telephone or internet connection.

In the market, there is a desire to promote and conduct multi-lingualmeetings via conference call, either replacing or extending the“in-person” meeting with a “virtual” meeting that includes participantsconnected from remote locations. A traditional conference call allowsall participants to enter into a common bridge and hear each other as ina “party line” telephone call. If participants on the bridge speakdifferent languages and cannot understand a common language,communication is quickly made impossible under this “party line” model.

In existing simplified models, in a conference call where participantsspeak different languages, the use of “consecutive” interpretation isoften contemplated. In this mode of operation, an interpreter isincluded as an additional participant in the conference on the bridgeand an agreement is reached to allow the interpreter time to interpret.When, for example, a delegate speaks in Spanish, she pauses after one ortwo sentences, and the interpreter repeats what she said in English. Thedelegate then resumes speaking briefly, and the process iterates for theduration of the entire conversation. When a delegate speaks in English,the interpreter waits for the delegate to pause, and then repeats whatwas said in Spanish. Everybody hears all of the Spanish and Englishutterances. This mode of interpretation is used, for example, when asports figure is a guest on a television show and only a handful ofquestions will be presented to the athlete.

This approach is very slow and tedious, and makes the dialogue much lessdynamic. While operating this system in two languages is a significantchallenge, it becomes completely unwieldy when multiple languages areinvolved.

A further complication is that, unlike the in-person meeting at theUnited Nations, where the participants and interpreters can bepositioned so they can see each other, over-the-phone meetings relyalmost exclusively on audio cues. Participants need to somehow be ableto glean the tone and demeanor of the speaker and to interrupt oneanother. They must avoid overrunning the interpreter or cutting eachother off. This can be virtually impossible.

What is needed is a conference call capability that allows forsimultaneous interpretation in two or more languages, without burdeningdelegates or interpreters with additional language constraints.Participants need to have a sense of context, so that they can yieldwhen someone else wants to talk. The management of the conference mustbe left primarily to automated systems, so that the participants canfocus on the topic of discussion and the interpreter(s) can devote theirfull concentration to their language duties.

Mr. David P. Frankel, the originator of the current invention, is anexpert in the field of audio conferencing. In 2006, Frankel invented anew multi-fidelity conferencing bridge that allows participants tobenefit from improved clarity and accuracy of conferencing bridges bydeveloping a way to use Voice-over-Internet Protocol technology (VoIP)or public switched telephone network (PSTN) where both narrow band andwideband technology can peacefully coexist. Users who dial into thebridge with a wideband enabled tool are not forced down into the lowerfidelity narrow band. Ultimately, this technology was patented as U.S.Pat. No. 7,986,644 (“Frankel I”). The content of Frankel I is herebyincorporated fully by reference as part of this application.

The next year, in 2007, Frankel improved conferencing bridges byinventing a new identity-based conferencing system where a bridge iscapable of recognition of the identity of the individual users, forexample to recognize a phone source identification number associatedwith the user. Through automatic recognition, the burden on the user ofthe conferencing system is alleviated as the user is able to access thesystem with less authentication information. This technology waspatented as U.S. Pat. No. 7,343,908 (“Frankel II”). The content ofFrankel II is hereby incorporated fully by reference.

The same year, in 2007, a different invention described in U.S. Pat. No.8,041,018 (“Wald”) was filed. Wald describes a conference bridge shownas FIG. 1 taken from the prior art where a main language circle is usedas described above where each of the participants (P1 to P7) isconnected to the main language circle. As part of this system, all ofthe interpreters (L1 to L3) must interpret between the main language andone other non-main language.

Wald is extremely limited in that it is rooted in the use of andconnection to a main language of all participants where all othersubsequent languages connect. This invention is not applicable tocomplex systems. For example, the United Nations operates with sixofficial languages, and the European Community has twenty-four; Waldsimply cannot be used by these users. In these institutions, not all ofthe meetings take place in all of the official languages but, to bepractical, any system deployed in such a multi-lingual environment mustbe more versatile in that it must be able to accommodate numerous activelanguages and must provide for various styles of interpretation.

In some cases, UN interpreters work in only one direction; in othercases they work bi-directionally (i.e., they interpret back and forthbetween two languages). Relay interpretation in these institutions isalso part of the standard operating procedure. The requirement that onelanguage be designated as “primary” or “main” or “base” or be used byall of the interpreters is not acceptable. Wald would not be appropriateas there is no willingness to designate certain participants as “secondtier” just because they do not speak the “primary language” of themeeting; in fact, it is critical in diplomatic conversations that thedifferent languages all be treated equally. The Wald system requiresthat the interpreters (L1 to L3) in FIG. 1 all speak a common “main”language. The system fails unless L1, L2 and L3 speak the same language.Further, relay interpretation will not necessarily go through just asingle language; there might be relay of Arabic to French to English,but also Chinese to English to Spanish. What is needed is a system ofinterpretation that can be used by any group or institution that isstructured to allow for a wide diversity of use without the need for a‘main’ language.

Partly to overcome some of the problems and limitations of Wald, in2011, Frankel invented a new technology for conferencing bridges thatallowed for the management of interpretation for users calling into thebridge and speaking different languages. The system as described used afloor control and acted as large dispatcher of the flow of speechbetween the different users and interpreters connected to the bridge.This technology was patented as U.S. Pat. No. 8,175,244 (“Frankel III”).

The system as shown at FIG. 2, taken from the prior art, namely, fromFrankel III, is fully incorporated herein by reference. Frankeldescribes how a series (1, 2, 3 . . . N) of users also called delegates10 are connected via a network 30 to a conference server 40 where thedifferent signals are processed. A plurality of interpreters (1, 2, 3 .. . N) 20 are also connected through a network 30 to the same server 40where data is also processed. FIGS. 3 and 4 taken from Frankel III, showthe flow of audio and/or video exchange when delegate 2 (D2) speaks asshown at FIG. 2, and when delegate 4 (D4) speaks as shown at FIG. 3. Inthese figures, four different bridges 48 a, b, c, d are shown each wherethe conversation evolves in a different language (Mandarin 48 a,Japanese 48 b, English 48 c and French 48 d). In this model, there is no“main” bridge or “main” language as described in Wald.

FIGS. 3 and 4 of Frankel III use arrow heads to show how the data, voiceand potentially image, are transmitted by the server 40. The Frankel IIIsystem shown at FIGS. 2, 3 and 4 allows for consecutive or simultaneousinterpretation. This flexible and modular system allows for multipleusers, delegates and interpreters, to connect to a server 40 in a widerange of configurations as needed by a client. To help understand thetechnology, the Microsoft® Corporation could desire to connect in asingle conference call the four main design teams of its internationalunits, these are located in Houston (English), Tokyo (Japanese), Taipei(Mandarin) and Montreal (French). The limited technology of Wald forcesthe organizers to define English as a “main” language and then requirethe help of three very specific interpreters, a Japanese/English, aMandarin/English and a French/English interpreter.

Frankel III is much more flexible, as described at FIGS. 2, 3 and 4.First, there is no requirement for any of the four languages to bedefined as “main.” Four different bridges would be set up in the systemand each would be linked using three different interpreters. Because thesystem is enabled once all four bridges are linked by any combination ofinterpreters, the choice of interpreters given to the system owner andoperator is made simpler. As shown at FIG. 4, a first set of acceptableinterpreters includes: (a) Japanese/Mandarin, (b) English/Japanese and(c) French/English. In fact, other sets of interpreters could also beused, such as: (a) Mandarin/French, (b) English/Japanese and (c)French/English; or even (a) Mandarin/English, (b) Japanese/English and(c) either English/French or Mandarin/Japanese. This is only one of thenumerous advantages of Frankel III over the prior art.

While Frankel III is described at FIGS. 2, 3 and 4 and incorporated aspart of the disclosure of this invention, it can be further improved. Inthe example shown at FIG. 4, the words of the Japanese delegate D4 arefirst interpreted by SI2 into English and then in turn the words areinterpreted by SI1 in French. Even with simultaneous interpretation, thedelay of the first interpretation is added to the delay of the secondinterpretation, resulting in a time lag. In a conversational setting asshown at FIG. 4, the delegates listening into the Mandarin and Englishbridges 48 b, 48 c will hear a feed delayed by 2-3 seconds while thedelegates listening into the French bridge 48 d will be delayed by 4-6seconds. The lag between two different bridges renders activeparticipation difficult. What is needed is a system and method designedto improve Frankel III capable of maintaining the numerous advantages ofthe system while allowing the participants to improve their overallexperience.

SUMMARY

The present invention relates to a new method and system for use of amulti-protocol conference bridge, and more specifically a newmulti-language conference bridge system and method of use wheredifferent cues, such as an attenuated voice of an originalnon-interpreted speaker is used to improve the flow of information overthe system.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain embodiments are shown in the drawings. However, it is understoodthat the present disclosure is not limited to the arrangements andinstrumentality shown in the attached drawings.

FIG. 1 is the fourth figure of U.S. Pat. No. 8,041,018 from the priorart.

FIG. 2 is the first figure of U.S. Pat. No. 8,175,244 from the priorart.

FIG. 3 is the second figure of U.S. Pat. No. 8,175,244 from the priorart.

FIG. 4 is the third figure of U.S. Pat. No. 8,175,244 from the priorart.

FIG. 5 is a diagrammatic representation of a system on which can beimplemented a multi-lingual conference bridge with cues.

FIG. 6 is an illustration of the system illustrating the currentmulti-lingual conference bridge with cues according to an embodiment ofthe present disclosure.

FIG. 7 is a diagrammatic representation of the fluxes of data andassociated fluxes of cues generated or relayed by the multi-lingualconference bridge with cues of FIG. 5.

FIG. 8 is a diagrammatic representation of an original speech, severalsimultaneous interpretations and an attenuated cue according to anembodiment of the present disclosure.

FIG. 9 is an illustration of the different steps associated with amethod of holding a multi-language conference using a multi-protocolconference bridge.

DETAILED DESCRIPTION

For purposes of promoting and understanding the principles disclosedherein, reference is now made to the preferred embodiments illustratedin the drawings, and specific language is used to describe the same. Itis nevertheless understood that no limitation of the scope of theinvention is hereby intended. Such alterations and further modificationsin the illustrated devices and such further applications of theprinciples disclosed and illustrated herein are contemplated as wouldnormally occur to one skilled in the art to which this disclosurerelates.

Portions of Frankel III already fully incorporated by reference arereproduced herein to help with the overall comprehension of this newinvention and how it may be implemented in one of many differentmulti-lingual conference bridges. For example, what is shown is how thecurrent invention can be implemented on a bridge as described in FrankelIII and shown at FIGS. 2, 3 and 4. As part of this description, twoconcepts are described. First, the user of a system that speaks andunderstands at least one language and has a need to communicate,understand and exchange with other users of the system who may speak orunderstand a different language; these “users” are also described as“delegates,” “speakers,” or “callers” but other terms of equal meaningcould be used. One of ordinary skill in the art of multi-lingualconference bridges will understand that, while several terms are used,what is contemplated and described is the concept of a person or even adevice such as a machine needing active interpretation of a conferenceor multi-participant communication. Further, the term “interpreter” isused as an individual capable of listening into a conversation spoken ina first language and capable of producing a semi-simultaneous new spokenstring of words in a second language. While one word is used, otherslike “live translator,” “simultaneous translator,” “multi-lingualindividual” or other expressions can be used. What is also contemplatedis that the different interpreters instead of being humans may be partlyor fully automated systems capable of interpretation. Also what iscontemplated under the term “language” is a protocol of communication,or information exchange, between two parties or users of themulti-lingual conference bridge. Currently, the inventor contemplates asthe best mode of operation of the technology the application to audiointerpretation and audio cues. While a language is generally recognizedas a spoken means of communication between two different groups, thefollowing invention could be extrapolated with time to much broaderconcepts when different needs arise and the technology can beextrapolated.

Computer program instructions may also be stored in a computer-readablememory that can direct a computer or other programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable memory produce an article of manufactureincluding computer-readable instructions for implementing the functionspecified in the flowchart block or blocks. The computer programinstructions may also be loaded onto a computer or other programmabledata processing apparatus to cause a series of operational steps to beperformed on the computer or other programmable apparatus to produce acomputer-implemented process such that the instructions that execute onthe computer or other programmable apparatus provide steps forimplementing the functions specified in the flowchart block or blocks.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions, andprogram instruction means for performing the specified functions. Itwill also be understood that each block of the block diagrams andflowchart illustrations, and combinations of blocks in the blockdiagrams and flowchart illustrations, can be implemented by specialpurpose hardware-based computer systems that perform the specifiedfunctions or steps, or combinations of special purpose hardware andcomputer instructions.

FIG. 2 illustrates an exemplary language-aware conferencing system(LACS) 8. As part of this system, delegates 10 and interpreters 20 areeach equipped with a telecommunications device (shown at FIG. 5). As anexample, the telecommunications device can be a conventional telephone,a computer with audio input/output capability, a smart phone, or somepurpose-built device. As a further example, the telecommunicationsdevice can be equipped with a display. In certain aspects, at least onedelegate 10 and/or interpreter 20 is equipped with both a telephone foraudio exchange as well as a computer with a display and controlcapability.

The telecommunications devices of the delegates 10 and the interpreters20 are in signal communication with a conference server 40 viaconnections 35. In certain aspects, the connections 35 are definedthrough one or more networks 30. As an example, the network(s) 30 can bethe conventional public telephone network, the Internet, a mobiletelephone network, a private network, or some combination of networksnow known or later developed.

In certain aspects, the conference server 40 includes the functionalelements of conventional conferencing systems, including audio bridges42 that mix the audio for selected groups of participants, detectors 44that sense inputs (such as keypad digits), and control logic 46 thatdetermines how the audio bridges mix the various inputs and outputs. Itis understood that the conference server 40 can include any number ofcomponents and sub-components for executing the methods disclosedherein. As an example, the conference server 40 can be implemented assoftware running on a conventional hardware platform (computer server)or a purpose-built hardware or a combination thereof.

FIG. 3 illustrates six of the delegates 10, identified individually asD1 through D6, and three of the interpreters 20, identified individuallyas SI1 through SI3, connected to conference server 40. As shown, two ofthe delegates D2, D3 speak English as a native language, two of thedelegates D5, D6 speak Mandarin, one of the delegates D1 speaks French,and one of the delegates D4 speaks Japanese. One of the interpretersspeaks French and English SI1, one of the interpreters SI2 speaksEnglish and Japanese, and one of the interpreters SI3 speaks Japaneseand Mandarin. Accordingly, the conference server 40 as shown, isconfigured with four bridges 48 a, 48 b, 48 c, 48 d, one for eachlanguage. While the technology allows for the conference server 40 to beconfigured for any number of languages and bridges, and where multipleserial interpretations can occur, the inventor discloses that under thebest mode of operation, it is preferable to limit the number of serialinterpretations to minimize the degradation of the quality of theinterpretation.

As shown in FIG. 3, the delegate D2 “has the floor” (or is acknowledgedas the speaker) and is shown with an underline, and a plurality of audiopaths 50 are configured accordingly. As shown, the arrows of the audiopaths 50 indicate a directional flow of the audio data/signal. With theexception of the delegate D2, all of the delegates 10 are in alisten-only mode, connected to their respective bridges 48. Audio fromthe delegate D2 is fed to the English bridge 48 c. The interpreters SI1,SI2 are both listening to the English bridge 48 c, wherein theinterpreter SI1 is speaking into the French bridge 48 d, and theinterpreter SI2 is speaking into the Japanese bridge 48 b. Theinterpreter SI3 is listening to the Japanese bridge 48 b and speakinginto the Mandarin bridge 48 a.

FIG. 4 shows the same set of delegates 10 and interpreters 20 as FIG. 3.As shown, the delegate D4 now has the floor, and the audio paths 50through the bridges 48 have been reconfigured accordingly. The delegateD2 is shown in a listen-only mode; the delegate D4 is shown speakinginto the Japanese bridge 48 b; and the interpreter SI2 is shownlistening to the Japanese bridge 48 b and speaking into the Englishbridge 48 c. It is understood that the audio paths 50 and bridges 48 canbe configured for any number of delegates 10 and interpreters 20speaking any number of languages.

The present methods and systems can be operational with numerous othergeneral purpose or special purpose computing system environments orconfigurations. Examples of well-known computing systems, environmentsand/or configurations that can be suitable for use with the systems andmethods include, but are not limited to, personal computers, servercomputers, laptop devices and multiprocessor systems. Additionalexamples include set top boxes, programmable consumer electronics,network PCs, minicomputers, mainframe computers, distributed computingenvironments that comprise any of the above systems or devices, and thelike. The systems and methods can be implemented in a public or privatenetwork and delivered as a cloud service.

A problem unique to multi-language bridge systems is the subsequent needfor multiple interpretation of the same speech. For example, using thediagram shown at FIG. 3, three bilingual interpreters SI1, SI2, and SI3are placed between the four bridges 48 a, 48 b, 48 c and 48 d. If thefloor is given to what is shown as delegate 6 (D6) who speaks inMandarin into bridge 48 a, before the French speaking-delegate hears theconversation from the French bridge 48 d, three interpretations willoccur serially. While one way to increase the speed of the flow ofcommunication between the different delegates, users or other partiesusing the system is to use simultaneous interpretation where thedifferent interpreters are competent enough to initiate interpretationas the initial speaker is still talking, what is shown in this inventionis the use of additional tools and methods to further increase the flowand speed of the system.

In addition to quality of information flowing in and out of a bridge,and the clarity of the information flowing in and out, one of the mostimportant aspects is the dynamic flow of the information as participantstalk and interpreters interpret. By dynamic flow we mean the capacity ofthe different users/Delegates D1 to D6 to receive the information,understand the information, process the information and be able torespond or act based on the information received. Conversations andconferences are generally improved as they become more fluid and thedynamic flow is increased. Participants in a more dynamic flowenvironment will feel more engaged and interested if they are able toquickly understand the information and respond without undue delays.

In an effort to better understand dynamic flow of conversation over abridge and the different time delays and other time incrementsassociated with overall bridging technology, time delays will bewritten, for example, using the Greek letter delta (Δ) to illustrate ashort increment and the letter T for time (ΔT). While this terminologyis used herein, one of ordinary skill in the art will recognize that anytype of terminology and numerology could be used.

When understanding a flow of conversation between multiple people, thereare multiple time delays ΔT that are created at different points in theoverall conference over a bridge. For example, FIG. 5 shows a system 100where a user 101 uses either a computer 104 or other device 108 such asa phone 110, a web enabled phone 111, a pad 112, or any other type ofcomputer device 113 to log into a server 102 where the bridge exists.The connection will be done, for example, over a network 103 such as theInternet or any wireless network. A second user of the bridge 105 willalso be connected and using a device 108 and, as shown here, eitherother users or interpreters 106 will also be connected to the bridge onthe server 102.

Inherent to the use of any technology is a series of technologicaldelays that can be understood as a sum of delays each associated withthe responsiveness of the system, the power of the server, the qualityof the connection, the quantity of equipment used, etc. In addition todelays that can be improved by improving the different technology used,there are software delays ΔT^(soft) associated with the management bysoftware on the server of the different bridges. Because of rapidtechnical advances in computer sciences, these software delays can bereduced. Finally, conversational delays ΔT^(conv) also exist and are themajor share of the total delay felt. For participants of any givenconference call over a bridge, the total delay felt ΔT^(T) will be a sumof these other delays, namely:ΔT ^(T) =ΔT ^(tech) +ΔT ^(soft) +ΔT ^(conv)

In cases where technological delays and software delays are very small,the overall total delay will be greatly improved as conversationaldelays are reduced. The current invention relates in part to theimprovement of the enjoyment and usefulness of different types ofconference bridges and other visual and oral systems as a result of thereduction of the total delay ΔT^(T) by a new dynamic of operation of theconversational flow and the interpretation flow that results in a newway to reduce the conversational delay ΔT^(conv).

As shown at FIG. 3, the system includes four different bridges 48 a, 48b, 48 c and 48 d with three interpreters SI1, SI2 and SI3 interpretingbetween the different bridges. In this scenario, for the French delegateD1 to understand the Mandarin delegate D6, three subsequent interpreterseven when simultaneous interpretation is used must hear a conversationand interpret into a second bridge.

The telecommunications devices of the delegates 10 and interpreters 20communicate with a language-aware conference server (LACS) 40 viaconnections 35 through one or more networks 30. The network could be theconventional public telephone network, or it could be the Internet, or amobile telephone network, or a private network, or some combination.

It is important to note that the invention does not normally requirespecial technology at the location of each delegate and interpreter;they can use the same technology that they use for conventionaltelephone and computer applications. However, the devices andconnections of the interpreters must be capable of full-duplex audiotransmission without interference between transmit and receivedirections; if the available technology does not support this over asingle connection, it can be accomplished with two independentconnections for an interpreter, and appropriate interface equipment atthe interpreter's location. It is a contemplated best mode that that thedelegates have two-way connections but the requirement for lack ofinterference is less stringent.

The conference server 40 comprises the functional elements ofconventional conferencing systems, including audio bridges that mix theaudio for selected groups of participants, detectors that sense inputs(such as keypad digits and audio activity), and control logic thatdetermines how the audio bridges mix the various inputs and outputs. Theconference server can be implemented as software running on aconventional hardware platform (computer server), or it could bepurpose-built.

The current inventor has discovered that the human brain can be givenseveral key types of information and cues associated with the originalspeech, an interpreted speech, or any other factor that can be made totravel in parallel of any consecutive and/or simultaneous interpretationto broadcast to a listener information about the conversation. Forexample, interpreters may hesitate or struggle with the conveyance ofintonations or inflections in a voice. This problem can be heightened ifseveral serial interpreters must interpret the same stream of data fromone bridge to the next. A listener's reaction time and focus may belowered if the listener does not know that the speaker is extremelyagitated or uses a tone that relays urgency and importance. Further,when a Delegate wishes to interrupt or question a currently-speakingDelegate, it is important that that speaking Delegate know this so thatshe can pause to allow the first Delegate to speak. But if she has towait for one or more interpretations before hearing any indication ofthe desire to interrupt, the conversation is delayed. Thus, importanttime in the overall ΔT^(conv) can be saved by providing audio cuesbeyond just passing the interpreters' speech.

The new system as shown at FIGS. 5, 6 and 7 is a centralized systemwhere each user is connected to that system by telephone or anInternet-based connection with audio, video, a combination thereof orany other data relay equipment. Each of the interpreters is assigned to(or select) a specific language that they will “listen from” and anotherthat they will “interpret into.”

First, if a person is connected to a specific bridge with a specificlanguage (e.g., Russian) and the current speaker is also speakingRussian, the feed will be simply that of the original feed. In thiscase, an interpreter assigned to listen to a different language andinterpret into the Russian bridge will be idle. As a different speakerinitiates in a different language, and the “listen from” bridge of theidle interpreter is filled with sound, the Russian interpreter is nowable to interpret the speech heard. Via a keyboard, a telephone keypador any other input system, a user/delegate can select any bridge andchange the bridge at any time. Thus, a delegate bilingual in French andRussian might start the meeting listening without interpretation. Butwhen another delegate starts speaking Spanish, the first delegate couldchoose to listen in French, and take advantage of the Spanish-to-FrenchSI.

SIs can change their language selection as well. This allows a single SIto play the role of, say, French-to-Spanish interpreter as well asSpanish-to-French interpreter. SIs can also elect to listen to anotherSI rather than to the delegates. This allows for “relay interpretation”described above where, for example, Russian is interpreted into Spanishby way of first being interpreted into French by one SI, and from thereinto Spanish by another SI.

The novel system is equipped to generate and manage the transfer to thedifferent participants on the system with “audio cues” or “visual cues”or any other “cues.” These cues, for example, allow a user/delegate toaccess information and data relating to either other participants ordelegates, and information relating to any of the interpreters as towhen they are speaking, or when they are stopping, to better manage theflow of the conversation. This is very important because, without audio,visual or other cues, participants depend only on audio feedback to knowwho is talking and when they can speak or interrupt.

Once cues are sent back to the different users of the system, whatresults is a dynamically flowing conversation where the multi-lingual“virtual meeting” experience mimics as closely as possible the physicalmeeting experience. Delegates and SIs can be located anywhere. Delegatescan listen and speak in their preferred language. The conversation flowsas it would for a physical meeting, without the slowdown and tedium ofconsecutive interpretations.

One more way to implement the prior art system shown at FIGS. 2 to 4 isto: (a) establish a separate conference call for each of the N languagesneeded for the call; as part of the system as shown, each of theseparate conference calls is referred to, for example, as a “bridge”;(b) instruct the users of the system that are speakers of a firstlanguage to connect to a specific bridge, and the speakers of a secondto an Nth language to connect to a second bridge, and so on for each ofthe N languages; (c) instruct an interpreter (simultaneous orconsecutive) of two of the languages to connect to the two bridges thatrelate to the languages he/she can interpret; this may be done using twoseparate data transfer lines like two telephones and associatedtelephone lines; and (d) instruct other interpreters with the neededskills to connect until all bridges are connected as shown at FIG. 6. Inone embodiment, the interpreters may wear two headsets with twomicrophones where one headset can cover the left ear and the otherheadset would cover the right ear and use his other connection.

Further, the method includes the step where (e) one of the interpretersis listening to a first language in his right ear, and interpreting intoa second language into the second mouthpiece or other similar device.The interpreter would have to “mute” the right mouthpiece, becauseotherwise his words would be fed back to the bridge of the firstlanguage and interfere with those listening to that bridge. What resultsis a semi-automated system where an interpreter must use two differentphones and lines. One of ordinary skill in the art will recognize thatthe instant invention offers significant benefits beyond the alternativeprimitive implementation.

While the above description is directed at a conference bridge relyingmostly on audio data, and the current state of the technology in thefield of multi-lingual conferencing strongly suggests the reliance ofaudio data, the inventor understands that the same technology can beextrapolated to the use of mostly non-audio data.

The same technology can be implemented on mostly video/audio platformslike Skype®. This interface allows the connection of audio and videofeeds by multiple simultaneous users. Using existing technology, if onehundred participants desire to join the conference, they will call intoa single bridge where it will be very difficult to manage theconference. To help group users in different language, users might setup physical locations (offices) where all of the Japanese speakers areinvited to come so the main conference will have a limited number ofactual users. Using the Frankel III technology, the LACS technologycould be integrated with the Skype® conferencing capability, wheremultiple bridges are set up for the multi-lingual meeting and each userwill be able to simply log into the bridge where only his/her languagewill be heard. Most of the time, when a user is talking in a differentlanguage, the voice of the interpreter will be heard. Using the Skype®analogy is only one of a large number of software platforms that can beused in addition to ordinary audio call systems to implement theinvention of Frankel III and the current invention.

What is described and shown at FIGS. 5 to 7 is a system that is animportant enhancement to all known multi-language conference systems,including, for example, the system described at FIGS. 2 to 4. The systemdescribed above eliminates the notion of separate conferences or bridgesand replaces it with a holistic, language-aware system that delivers anefficient, seamless, flexible and reliable experience to users likedelegates and interpreters alike, using a dynamic, flexible andindividually customizable computer-controlled determination of thedifferent data streams.

In this new method, the users, also known as delegates 110 andinterpreters 120, connect to our “language-aware” conferencing system(LACS) 140, also shown to include a conference server 150. The LACS 140associates each delegate, here delegate 1 to delegate N with a listeningpreference based on specific input from the delegate, or based on someother selection mechanism. The delegate could select the listeningpreference using an IVR system when first connecting, or he could dialsome code on his telephone keypad (e.g., *41 for Spanish, *42 forMandarin, or *40 for no interpretation).

In other embodiments, the listening preference is made based on apersonal identification identifier associated with the country fromwhich he was calling (Spanish if calling from Mexico, Spain, orArgentina), determined either by the number where he called from or theaccess number of the conference bridge dialed when a phone number isavailable in each of several different countries to connect to thesystem. Other features of the system are described and incorporated byreference from Frankel III.

The server 150 is a system that includes a control logic module 151, acues detection module 152, also described as a detector for audio and/orother types of tones. The server 150 also may augment one or a pluralityof audio bridges (shown at FIG. 7) with selective attenuation or othersystem such as an attenuation module 153 for providing cues within thedifferent feeds in the system.

One novel element of the current invention is to provide audio cues inaddition to the audio/video streams of information described above andin Frankel III to the users/delegates so that a party using the systemis aware of the interpretation process and can sense when otherdelegates are speaking, the tone of the different actors using thesystem, the inflections, and the redundancy and importance of the streamunder interpretation.

Of great importance is to note that during the process of simultaneousinterpretation, as in any interpretation, there are natural delays andsilences created by the interpreter as they find the right concepts andwords to interpret. The cues transmitted alongside other normalinterpretations will allow a user to recognize when the interpreter issilent for several seconds even though a delegate in the foreignlanguage continues to speak. Similarly, in cases where the interpretersneed a few seconds to finish interpreting after a delegate or user hasstopped speaking, the use of cues will allow the listener to manage theresponse being prepared.

In one embodiment, one family of cues is the use of an attenuated levelused as a background to be layered on top of the interpretation as afeed to a delegate and/or an interpreter. The use of a background lowerlevel feed can in some cases not interfere with comprehension of theprimary audio source, i.e., the interpretation that can be understood bythe user/delegate.

The server 150 includes a system that generally is set up not to “echo”audio data back to its source. For example, if inputs from users A, Band C are being mixed by the server 150, the user A will not hear back amerged feed that includes its own audio but will only hear B plus C, Bwill hear A plus C, and C will hear A plus B or whatever is the casedescribed below as managed by the control logic module 151.

As shown at FIG. 6, the bridges of FIG. 7 are capable of attenuatingspecific input signals to varying degrees as managed by the attenuationmodule 153. For example, in the above example where user A listens touser B and C, this communication C could be set for attenuation by afixed level; in one embodiment the second channel attenuated and used asa cue is lowered by approximately 12 decibels. As described above, userA would hear user B at a normal volume and would hear user C softly inthe background. A connection to a bridge can also be unidirectional, sothat a party supplying input to a bridge does not hear the output fromthe bridge (or vice versa).

The module for the detection of audio and/or tones 152 is designed to beflexible and capable of adaptation based on the type of technology inwhich the system is adapted to be implemented. For example, in the caseof a multi-lingual audio conference system, most of the data exchangedis in the form of voice and the detector module 152 is programmed fordetection of different tones within the voice. For example, the detectormodule 152 detects silences, pauses, extreme verbal responses, certainkey words, small electronic sounds, keyboard or numerical tones of adial phone, names, or even when too many voices overlap and result inpotential confusion. In the case of other, text based systems, thedetector module 152 can recognize key words typed, written requests, orother data. The same logic of data modulation and detection can beapplied to any system based on the technology.

FIG. 7 shows a system 150 where six delegates (D1 through D6) 110, andthree interpreters (SI1 through SI3) 120 are connected to the conferenceserver 140 using a system as shown at FIG. 5. Here, two of the delegates(D2, D3) speak English, two delegates speak Mandarin (D5, D6), onedelegate (D1) speaks French, and finally one delegate (D4) speaksJapanese. Interpreter (SI1) speaks French and English, a secondinterpreter (SI2) English and Japanese, and a last and third interpreter(SI3) speaks Japanese and Mandarin. In FIG. 7, the conference server 140is configured with four active bridges 148 a, 148 b, 148 c and 148 d,one for each language.

At FIG. 7, delegate D2, as shown by the underline, is the activeparticipant and, for example, is speaking on his cell phone. The darkarrows show how the control logic module 151 of the conference server150 relays the different voices. The dashed arrows also described asbroken lines indicate how the selective attenuation module 153 using thedetector module 152 will send and mix audio with attenuation to thedifferent users and/or bridges.

In Frankel III, as shown at FIG. 2, the delegate D2 speaks into theEnglish bridge 48 c as shown by the arrow. Both SI1 and SI2, whounderstand English, hear the bridge 48 b and interpret into bridges 48 band 48 d, respectively. SI3 listens to bridge 48 a and each of thedelegates listens to their respective bridges 48 a, 48 b, 48 c, and 48d.

FIG. 7 shows the different configuration of the current embodiment whereeach of the delegates are connected to listen to their respectivebridges 148 a, 148 b, 148 c and 148 d but where the connection of thedifferent interpreters SI1, SI2 and SI3 differs. In this example, SI3,instead of listening to the Japanese bridge 148 b, is connected directly171 to listen to the words spoken by the interpreter SI2. For example,SI3 does not need to hear D2 speak and is only focused on theinterpretation of the information in Japanese, namely, the voice of SI2.While this is suggested as a best mode, what is contemplated is the useof any system without this direct link and with SI3 able to connectdirectly to 148 b.

The voice of D2 is sent directly to the bridge 148 c, and is then sent172 via attenuation to the Mandarin, Japanese and French bridges 148 a,148 b and 148 d, respectively. The D2 voice is then heard in thebackground at a lower level by those listening to these bridges. Thisallows the other delegates (D1, D3, D4, D5 and D6) all connected to abridge to know that D2 is still speaking even when their interpreters(SI1, SI2 or SI3 are momentarily silent). They will know instantly viathe attenuated cue not to take their turns speaking until D2 isfinished.

In a subsequent embodiment, the voices of the interpreters SI1 and SI2interpreting from English into either Japanese or French (SI2 and SI1,respectively) are fed back quietly into the English bridge 173, 174.This allows the speaking delegate D2, and the non-speaking Englishdelegate D3, to know that the interpretation is taking place. In caseswhere the delegate D4 has the floor, the series of bridges would bereconfigured accordingly.

In another embodiment, a control panel 180 as shown at FIG. 5 can beused to display the control logic module information 151 or any othertype of information in a web browser operating in HTML or any otherprotocol and connected over the Internet or some other network to theLACS. This control panel can, for example, show the identity (name,telephone number, or other) of all of the participants in the meeting,and for each indicate what bridge 148 a, 148 b, 148 c or 148 d in theabove example is being used. The control panel 180 can also show thestatus of interpretation of each SI and use other types of codes andcues to help identify the process, and even the presence and identity ofthe SIs.

In yet another embodiment, the system 140 supports more than one SI perlanguage bridge 148 a, 148 b, 148 c or 148 c. Multiple SIs per bridge isused to help SIs with fatiguing simultaneous interpretation. The systemallows for several SIs to be assigned to a single bridge, for exampleSI1 in FIG. 7 would in fact be two or more incoming calls eitherside-by-side at the same location or managed at two different locations.In the case of different locations, the control logic 151 as shown atFIG. 16 manages the interpreter 120 using some type of handoff from oneto another, which can be by any number of means, for example analogousto those presented above for other functions. One SI for a bridge willbe given an active status and will be able to talk into the interpretedbridge and the second SI may be on standby in a listen-only mode. Thecontrol of the control logic 151 can be made automatically, by a controlpanel 180 or by any other means either at the system 140 or by the SIitself (not shown).

As part of the control logic 151 shown at FIG. 6 and as describedlogically at FIG. 7, three different simultaneous interpreters can eachbe capable of interpretation between two languages. In one example allthe SIs may share one common language but this is not a requirement ofthe invention (e.g., the SI3 capable of Japanese to Mandarininterpretation can be replaced with an English to Mandarininterpretation as long as SI3's input is bridge 148 c and not 148 b orSI2). In one embodiment, each interpreter understands at least twolanguages and is associated with (at least) two bridges.

The control logic 151 assigns interpreters starting with those speakingthe same language as the delegate that has the floor, and then iterateassignments so that all delegates are served. For example, a meetingmight include English-French (A), French-Dutch (B), Dutch-German (C) andGerman-Spanish (D) interpreters. When someone speaking Dutch has thefloor, interpreters B and C are assigned to listen to the Dutch bridge,since they can both speak the language of the delegate that has thefloor. Interpreter A listens to the French bridge, and interpreter Dlistens to the German bridge. Each interpreter speaks into his “other”bridge (that is, the bridge to which he is not listening). When aGerman-speaking delegate takes the floor, the connections areautomatically reconfigured or they are configured according to inputsfrom the Interpreters. In this example, words of an English-speakingdelegate conversing with a Spanish-speaking delegate would beinterpreted four times before reaching the other party.

In another embodiment, each interpreter may understand three or morelanguages. Even more complex arrangements are possible. An interpretermight be tri-lingual, in which case he will be assigned listening andspeaking bridges according to an algorithm that minimizes the number ofsequential interpretations required while still ensuring that everydelegate hears the interpreted audio. The interpreter may hearannouncements telling him what language to interpret from and to.However, even with interpreters capable of operating between three ormore languages, a meeting with delegates speaking N languages mayrequire N−1 interpreters if all delegates are to hear (near)simultaneous interpretations (i.e., the tri-lingual interpreter can onlyinterpret to a single language at a time).

In addition to the other digital uses described above, the system 140can be used in conjunction with a physical meeting, in any of severalways. Each group of like-language speakers at the physical meeting canbe treated as an additional delegate on the teleconference, and can beconnected via appropriate “patches” between the in-house audio systemand a telephonic or VoIP link. So, for example, the English speakers atthe physical meeting would have their microphone and speaker connectionsmerged and fed to the English bridge over one connection; likewise forthe Spanish speakers, and so on. Alternatively, the delegates at thephysical meeting can have individual connections to the system 140, orcan share audio connections (amongst those speaking the same language)but have individual buttons or other means to request and relinquish thefloor. The SIs can also be located at a physical meeting venue and“shared” with the system users, by means of independent audioconnections between each SI and the system 140. Other audio options andoptimizations are also possible.

SIs can be scheduled for work independent of the system 140, or thesystem can incorporate a database of SIs according to language abilityand connection information. As part of the logic 151, the system 140 canconnect the different needed SI automatically “on demand” according tothe language selections made by delegates.

In yet another embodiment, users can be permitted to move betweenbridges (i.e., between different languages). For example, aSpanish-speaking delegate that also understands English might prefer tolisten to a meeting in English so that he can hear the intonation andemotion in the English speaker's voice, and doesn't want to be subjectto interpretation when someone is speaking in English. However, thisdelegate fluent in Spanish but conversant in English may be morecomfortable speaking in his/her native language of Spanish into aSpanish bridge. These preferences can be handled automatically by thesystem 140, so, in our example, when our delegate had the floor (i.e.,an arrow leads out of his box as shown at FIG. 7), the delegate wouldsend voice to the Spanish language bridge. Such switching couldalternatively be done manually by the delegate using an input meansdescribed earlier.

Similarly, the system 140 can include a bridge called or referred to asa floor bridge. Listeners in this bridge would hear whoever had thefloor, in their native language. They would not get the benefit ofinterpretation. The floor bridge can be recorded to capture the originalaudio of all delegates and any attenuation or cue as desired. Otherbridges can also be recorded to capture the results of interpretation.

Usually, in most conferences what is encouraged is where only onedelegate has the floor at any given time, and all other delegates havethe respect to remain silent. However, the system 140 can be made toallow for overlap of different speakers on the floor based onprogramming preferences. Delegates may be allowed to speak over eachother or even over an SI, interrupting the current speaker at least tothe point that the speaker knows he should yield when convenient sosomeone else can talk. In a corporate setting, the CEO of thecorporation can be given floor preferences and allowed to enter and usethe floor when desired.

The system 140 can capture an audio recording from a selected bridge orall bridges. The audio for a particular bridge, or for all bridges, maybe transcribed using human transcriptions or automated speechrecognition technology and/or displayed or given to users. The system140 can be used together with web conferencing or can integrate webconferencing, so that delegates can view, for example, a slidepresentation, documents or other graphics. If desired, the presentationcould be tailored for each language.

FIG. 8 illustrates how the different flows of conversation can beimproved using attenuated speech. Illustrated on a time bar 200 is how aconference with multiple interpretation evolves. Illustrated on thisfigure is a first delegate D2 speaking two different sentencesillustrated by two arrows. Between both sentences, the delegate includesa gap in the speech but will resume several seconds later unlessinterrupted by one of the other delegates on the line. Illustrated byΔ1, the Japanese SI2 interprets simultaneously the two sentences. TheMandarin interpreter SI3 who is connected to SI2 will then alsointerpret the Japanese as fast as possible, creating a longer Δ2 delay.A person like D5 listening at FIG. 7 to the Mandarin bridge 148 a maychime in the moment the interpretation ends. As shown by the diagram,the end of the first Mandarin sentence 201 may arrive later in time thanthe start of the second sentence 202 spoken by the delegate D2.

Illustrated by dotted arrows 203 is an attenuated voice used as a cuerelayed into both the Japanese bridge 148 b and the Mandarin bridge 148a as shown. The delegate D5 will hear the Mandarin interpretation butwill, in the background, hear the start of the new D2 sentence. This cueis very helpful to guide the delegate D5 into coordinating his response.In this example, a gap exists between the first and the second sentence.The SI2 interpreting in Japanese as shown at FIG. 7.

FIG. 9 illustrates a method 500 of holding a multi-language conferenceusing a multi-protocol conference bridge, the method 500 consisting ofthe steps of requesting 501 and receiving 503 from each of a pluralityof users 501 a language preference for holding a conference, setting up504 on a conference server a plurality of audio bridges as shown at FIG.7 each for one of the language preferences of the plurality of users501, selecting 506 a plurality of interpreters 505 each capable ofunderstanding at least two of the languages of the audio bridges on theconference server, connecting 507 each of the plurality of users eachusing one of a plurality of first devices to the conference server on aserver connected to a network by matching the language preference ofeach user with the language in each of the audio bridges, connectingeach of the plurality of interpreters 508 to at least two of the audiobridges based on the languages understood by the interpreter, allowing509 one of the users to speak into the connected audio bridge of theuser's language preference, and relaying 511 by the conference serverthe speech of the user in the user's language preference into another ofthe audio bridges.

In further embodiments, the step of relaying the speech into anotheraudio bridge is first attenuated 510 in audio level by the conferenceserver, and also where one of the plurality of interpreters is connected512 only to one of the audio bridges and to another of the plurality ofinterpreters based on the languages understood by the interpreter. Alsothe step of connecting each of the plurality of interpreters to at leasttwo of the audio bridges based on the languages understood by theinterpreter includes the step of allowing the interpreter to receiveaudio from a first audio bridge and send audio to a second audio bridge,or wherein the audio sent to the first audio bridge is attenuated 513wherein the voice of the interpreter is sent to both the first and thesecond audio bridges.

As described above in great detail, what is disclosed is ateleconferencing system for simultaneous interpretation in a pluralityof languages that includes an audio connection for each of a pluralityof delegates, each delegate having a particular listening language, atwo-way audio connection for each of one or more interpreters, eachinterpreter being identified as having a listening language and aspeaking language, the two languages being different, a plurality ofaudio bridges for carrying out the following steps, each bridge capableof accepting inputs from one or more connections, mixing said inputswith specified levels of attenuation, and providing the resulting outputto one or more connections without echoing the audio received from aparticular connection back to that same connection.

The system also includes a means to configure the audio bridges suchthat each interpreter separately receives audio from all delegates,transmits audio to those delegates with a listening language matchingthe interpreter's spoken language for mixing without attenuation, andtransmits audio to those delegates with a listening language matchingthe interpreter's listening language for mixing with attenuation, suchthat the voice of the interpreter is heard softly and delegate voice(s)dominate, a means for each delegate to transmit audio to all bridges,and a means to separately detect when each interpreter is speaking and,when said speech is detected, to cause any bridge receiving thatinterpreter's speaking language to attenuate audio inputs from delegatessuch that delegates are heard softly and the interpreter's voicedominates.

In alternate embodiments, the audio connections are made over one ormore wide-area networks capable of connecting physically distinctlocations, one of the interpreters can either change the listeningand/or speaking language or change the listening language, one of thedelegates can be identified as listening without interpretation, andwill receive audio from all delegates without attenuation, aninterpreter can be assigned to receive audio either directly fromanother interpreter or from two or more interpreters, the change inlanguage is made using either keys on a telephone keypad or keys on acomputer workstation, at least one audio connection is selected from agroup consisting a conventional telephone network, and avoice-over-Internet protocol, and the attenuation of the level of thevoice of an interpreter is adjustable, and a delegate is a group ofindividuals all with the same listening language.

Finally, what is also described above is a multi-protocol conferencebridge system with a plurality of first devices where each of the firstdevices is used by one of a plurality of users for connection of each ofthe first devices over a network to a server, where each of the firstdevices includes a two-way connection for exchanging data between theusers and the server in one of many protocols of communication, theprotocol used being understood by the user, a plurality of a seconddevice where each of the second devices is used by one of a plurality ofinterpreters for connection to the server, where the second deviceincludes a two-way connection for exchanging data between each of theinterpreters and the server in more than one protocol of communication.

In other embodiments, the protocols of communication are differentlanguages, the bridges are audio bridges each in a different language,the users are conferences each in need of understanding the conferencein their own preferred language, and each interpreter is capable ofsimultaneous interpretation between two languages, the cue is anattenuated audio communication in a first language sent into a bridge tooverlap with a different language, the first and second devices arephones, the network is a wireless phone network, and wherein at leastone of the second devices instead of being connected between twodifferent bridges, is connected between a second interpreter and abridge.

It is understood that the preceding is merely a detailed description ofsome examples and embodiments of the present invention and that numerouschanges to the disclosed embodiments can be made in accordance with thedisclosure made herein without departing from the spirit or scope of theinvention. The preceding description, therefore, is not meant to limitthe scope of the invention but to provide sufficient disclosure to oneof ordinary skill in the art to practice the invention without undueburden.

What is claimed is:
 1. A teleconferencing system for simultaneousinterpretation in a plurality of languages, comprising: an audioconnection for each of a plurality of delegates, each delegate having aparticular listening language; a two-way audio connection for each ofone or more interpreters, each interpreter being identified as having alistening language and a speaking language, the two languages beingdifferent; a plurality of audio bridges for carrying out the followingsteps, each bridge capable of accepting inputs from one or moreconnections, mixing said inputs with specified levels of attenuation,and providing the resulting output to one or more connections withoutechoing the audio received from a particular connection back to thatsame connection; means to configure the audio bridges such that eachinterpreter separately: receives audio from all delegates; transmitsaudio to those delegates with a listening language matching theinterpreter's spoken language for mixing without attenuation; andtransmits audio to those delegates with a listening language matchingthe interpreter's listening language for mixing with attenuation, suchthat the voice of the interpreter is heard softly and delegate voice(s)dominate; means for each delegate to transmit audio to all bridges; andmeans to separately detect when each interpreter is speaking, and, whensaid speech is detected, to cause any bridge receiving thatinterpreter's speaking language to attenuate audio inputs from delegatessuch that delegates are heard softly and the interpreter's voicedominates, where the audio connections are made over one or morewide-area networks capable of connecting physically distinct locations.2. The system of claim 1, wherein one of the interpreters can change thelistening and/or speaking language and/or one of the delegates canchange the listening language.
 3. The system of claim 2, wherein thechange in language is made using either keys on a telephone keypad orkeys on a computer workstation.
 4. The system of claim 1, wherein one ofthe delegates can be identified as listening without interpretation, andwill receive audio from all delegates without attenuation.
 5. The systemof claim 1, wherein an interpreter can be assigned to receive audioeither from delegates or directly from another interpreter.
 6. Thesystem of claim 1, where at least one audio connection is selected froma group consisting of a conventional telephone network and avoice-over-Internet protocol.
 7. The system of claim 1, wherein eitherthe degree of attenuation of the voice of an interpreter and/or delegateis adjustable.
 8. The system of claim 1, wherein at least one delegateis a group of individuals all with the same listening language.